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1. Introduction 

The benchmarking of optimization software has recently gained considerable visibility. 
Hans Mittlemann's 1 14 1 work on a variety of optimization software has frequently un- 
covered deficiencies in the software and has generally led to software improvements. 
Although Mittelmann's efforts have gained the most notice, other researchers have been 
concerned with the evaluation and performance of optimization codes. As recent exam- 
ples, we cite 1 1 1213 14161 151191 . 

The interpretation and analysis of the data generated by the benchmarking process 
are the main technical issues addressed in this paper. Most benchmarking efforts involve 
tables displaying the performance of each solver on each problem for a set of metrics 
such as CPU time, number of function evaluations, or iteration counts for algorithms 
where an iteration implies a comparable amount of work. Failure to display such tables 
for a small test set would be a gross omission, but they tend to be overwhelming for 
large test sets. In all cases, the interpretation of the results from these tables is often a 
source of disagreement. 

The quantities of data that result from benchmarking with large test sets have spurred 
researchers to try various tools for analyzing the data. The solver's average or cumula- 
tive total for each performance metric over all problems is sometimes used to evaluate 
performance I1I4I6I . As a result, a small number of the most difficult problems can 
tend to dominate these results, and researchers must take pains to give additional in- 
formation. Another drawback is that computing averages or totals for a performance 
metric necessitates discarding problems for which any solver failed, effectively biasing 
the results against the most robust solvers. As an alternative to disregarding some of the 
problems, a penalty value can be assigned for failed solver attempts, but this requires 
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a subjective choice for the penalty. Most researchers choose to report the number of 
failures only in a footnote or separate table. 

To address the shortcomings of the previous approach, some researchers rank the 
solvers I 4I6I17I191 . In other words, they count the number of limes that a solver comes 
in fcth place, usually for k — 1,2,3. Ranking the solvers' performance for each problem 
helps prevent a minority of the problems from unduly influencing the results. Informa- 
tion on the size of the improvement, however, is lost. 

Comparing the medians and quartiles of some performance metric (for example, the 
difference between solver times |4|) appears to be a viable way of ensuring that a mi- 
nority of the problems do not dominate the results, but in our testing we have witnessed 
large leaps in quartile values of a performance metric, rather than gradual trends. If only 
quartile data is used, then information on trends occurring between one quartile and the 
next is lost; and we must assume that the journey from one point to another proceeds 
at a moderate pace. Also, in the specific case of contrasting the differences between 
solver times, the comparison fails to provide any information on the relative size of the 
improvement. A final drawback is that if results are mixed, interpreting quartile data 
may be no easier than using the raw data; and dealing with comparisons of more than 
two solvers might become unwieldy. 

The idea of comparing solvers by the ratio of one solver's runtime to the best 
runtime appears in [2|, with solvers rated by the percentage of problems for which a 
solver's time is termed very competitive or competitive. The ratio approach avoids most 
of the difficulties that we have discussed, providing information on the percent improve- 
ment and eliminating the negative effects of allowing a small portion of the problems to 
dominate the conclusions. The main disadvantage of this approach lies in the author's 
arbitrary choice of limits to define the borders of very competitive and competitive. 

In Section 13 we introduce performance profiles as a tool for evaluating and com- 
paring the performance of optimization software. The performance profile for a solver 
is the (cumulative) distribution function for a performance metric. In this paper we use 
the ratio of the computing time of the solver versus the best time of all of the solvers as 
the performance metric. Section[3]provides an analysis of the test set and solvers used 
in the benchmark results of Sections |4] and [5] This analysis is necessary to understand 
the limitations of the benchmarking process. 

Sections |4] and [5] demonstrate the use of performance profiles with results |9| ob- 
tained with version 2.0 of the COPS |7| test set. We show that performance profiles 
eliminate the influence of a small number of problems on the benchmarking process 
and the sensitivity of results associated with the ranking of solvers. Performance pro- 
files provide a means of visualizing the expected performance difference among many 
solvers, while avoiding arbitrary parameter choices and the need to discard solver fail- 
ures from the performance data. 

We conclude in Section [6] by showing how performance profiles apply to the data 
1141 of Mittelmann for linear programming solvers. This section provides another case 
study of the use of performance profiles and also shows that performance profiles can 
be applied to a wide range of performance data. 
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2. Performance Evaluation 

Benchmark results are generated by running a solver on a set V of problems and record- 
ing information of interest such as the number of function evaluations and the comput- 
ing time. In this section we introduce the notion of a performance profile as a means to 
evaluate and compare the performance of the set of solvers S on a test set V. 

We assume that we have n s solvers and n p problems. We are interested in using 
computing time as a performance measure; although, the ideas below can be used with 
other measures. For each problem p and solver s, we define 



If, for example, the number of function evaluations is the performance measure of in- 
terest, set t p>s accordingly. 

We require a baseline for comparisons. We compare the performance on problem p 
by solver s with the best performance by any solver on this problem; that is, we use the 

performance ratio 



We assume that a parameter vm > r p . s for all p, s is chosen, and r p s = tm if and only 
if solver s does not solve problem p. We will show that the choice of Tm does not affect 
the performance evaluation. 

The performance of solver s on any given problem may be of interest, but we would 
like to obtain an overall assessment of the performance of the solver. If we define 



then p s (r) is the probability for solver s € S that a performance ratio r PjS is within a 
factor r G M of the best possible ratio. The function p s is the (cumulative) distribution 
function for the performance ratio. 

We use the term performance profile for the distribution function of a performance 
metric. Our claim is that a plot of the performance profile reveals all of the major per- 
formance characteristics. In particular, if the set of problems V is suitably large and 
representative of problems that are likely to occur in applications, then solvers with 
large probability p s (r) are to be preferred. 

The term performance profile has also been used for a plot of some performance 
metric versus a problem parameter. For example, Higham 1121 pages 296-297] plots 
the ratio 7/||yl||i, where 7 is the estimate for the li norm of a matrix A produced 
by the LAPACK condition number estimator. Note that in Higham's use of the term 
performance profile there is no attempt at determining a distribution function. 

The performance profile p s : R h- > [0, 1] for a solver is a nondecreasing, piecewise 
constant function, continuous from the right at each breakpoint. The value of p s (\) 
is the probability that the solver will win over the rest of the solvers. Thus, if we are 
interested only in the number of wins, we need only to compare the values of p s (l) for 
all of the solvers. 



t. 



computing time required to solve problem p by solver s. 



mhi{tp iS : s £ S} 
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The definition of the performance profile for large values requires some care. We 
assume that r p s £ [1, tm] an d that r p>s = rjvf only when problem p is not solved by 
solver s. As a result of this convention, p s {ru) = 1> an d 

Ps = lim_ M T ) 

is the probability that the solver solves a problem. Thus, if we are interested only in 
solvers with a high probability of success, then we need to compare the values of p* 
for all solvers and choose the solvers with the largest value. The value of p* can be 
readily seen in a performance profile because p s flatlines for large values of r; that is, 
p s (r) = p* s for r G [r s , r M ) for some r s < r M . 

An important property of performance profiles is that they are insensitive to the 
results on a small number of problems. This claim is based on the observation that if p s 
and p s are defined, respectively, by the observed time sets t PjS and t PjS , where 

i P ,s = t p , s , P6P\M, 

for some problem q 6 V, then f PjS = r p>a forp G V \ {q}. Since only the ratio r gjS 
changes for any s G S, 

\p s {r) - Ps (t)\ < — , TER, 
n p 

for s G S. Moreover, /S s (r) = p s (t) for r < min{?' g iS , r qtS } or r > max{r 9 . s , f q s }. 
Thus, if rip is reasonably large, then the result on a particular problem q does not greatly 
affect the performance profiles p s . 

Not only are performance profiles relatively insensitive to changes in results on a 
small number of problems, they are also largely unaffected by small changes in results 
over many problems. We demonstrate this property by showing that small changes from 
r p s to f p s result in a correspondingly small L\ error between p s and p s . 

Theorem 2.1. Let ri and fi for 1 < i < n p be performance ratios for some solver, and 
let p and p be, respectively, the performance profiles defined by these ratios. If 

\r l ~f i \<e, 1 < i < n p (2.1) 

for some e > 0, then 

/oc 
\p{t)-p{t)\ dt<e 

Proof. Since performance profiles do not depend on the ordering of the data, we can 
assume that {r,} is monotonically increasing. We can reorder the sequence {fi} so that 
it is also monotonically increasing, and i2.1\ still holds. These reorderings guarantee 
that p(t) = i/rip for t G [r^, ri + i), with a similar result for p. We now show that for 
any integer k with 1 < k < n p , 



f k \p(t)-p(t)\ dt<k(—) , 
Ji \n p J 



(2.2) 
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where s k = max(rfc, fk), and 

\Ti — f%\ < e, 1 < i < fc, and fi — ri, k < i < n p . (2.3) 

The proof is completed when k = n p . 

The case k = 1 follows directly from the definition of a performance profile, so 
assume that (12. 2> holds for any performance data such that (12. 3> holds. We now prove 
that i2.2\ holds for k + 1 by proving that 

/ + \p(t)-p(t)\ dt< ( — ) , /c<n p . (2.4) 

Together, ( 12. 2> and ( 12. 4> show that ( 12. 2t holds for fc + 1. 

We present the proof for the case when f k < r k . A similar argument can be made for 
r k < f k . If rfc < r k then Sfc = rfc and f k < ?'fe < r k +i- The argument depends on the 
position of fk+i and makes repeated use of the fact that p(t) = k/n p for t 6 [rfc, rfc+i), 
with a similar result for p. 

If r k+ i < f k +i then p(t) = p(t) in [r fc ,r fe+ i). Also note that \p(t) - p(t)\ = l/n p 
in [r k +i, fk+i)- Hence, (|23 holds with s k+1 = f k+1 . 

The case where f k +\ < r k +i makes use of the observation that f j = > r^+x for 
i > k + 1. If r k < f k +i < r k+1 , then p(t) = p(t) in [r k ,f k+1 ), and - p(t)\ = 
1/iip in [f k +i, r k +i). Hence, i2A\ holds. On the other hand, if f k < f k +\ < r k , then 
we only need to note that \p(t) — p(t)\ — l/n p in [r k , r k+ i) in order to conclude that 
d2~4l holds. 

We have shown that ( 12. 21 holds for all integers k with 1 < k < n p . In particular, 
the case k = n p yields our result since p(t) = p(t) for t S [s„ p , cxo). □ 



3. Benchmarking Data 

The timing data used to compute the performance profiles in Sections@]and[5]are gener- 
ated with the COPS test set, which currently consists of seventeen different applications, 
all models in the AMPL 1 10 1 modeling language. The choice of the test problem set V is 
always a source of disagreement because there is no consensus on how to choose prob- 
lems. The COPS problems are selected to be interesting and difficult, but these criteria 
are subjective. For each of the applications in the COPS set we use four instances of the 
application obtained by varying a parameter in the application, for example, the number 
of grid points in a discretization. Application descriptions and complete absolute timing 
results for the full test set are given in (9) . 

Section |4] focuses on only the subset of the eleven optimal control and parameter 
estimation applications in the COPS set, while the discussion in Section [5] covers the 
complete performance results. Accordingly, we provide here information specific to 
this subset of the COPS problems as well as an analysis of the test set as a whole. 
Table 13. ll gives the quartiles for three problem parameters: the number of variables n, 
the number of constraints, and the ratio (n — n e ) / n, where n e is the number of equality 
constraints. In the optimization literature, n — n e is often called the degrees of freedom 
of the problem, since it is an upper bound on the number of variables that are free at the 
solution. 
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Table 3.1. Problem data for COPS test set 









Full COPS 




COPS subset 


mm 


<?i 


92 


93 


max 


mm 


91 


92 


93 


max 


Num. variables 


48 


400 


1000 


2402 


5000 


100 


449 


899 


2000 


4815 


Num. constraints 





150 


498 


1598 


5048 


51 


400 


800 


1601 


4797 


Degrees freedom 





23 


148 


401 


5000 





5 


99 


201 


1198 


Deg. freedom (%) 


0.0 


1.0 


33.2 


100.0 


100.0 


0.0 


0.4 


19.8 


33.1 


49.9 



The data in Table 13. ll is fairly representative of the distribution of these parameters 
throughout the test set and shows that at least three-fourths of the problems have the 
number of variables n in the interval [400, 5000]. Our aim was to avoid problems where 
n was in the range [1, 50] because other benchmarking problem sets tend to have a pre- 
ponderance of problems with n in this range. The main difference between the full COPS 
set and the COPS subset is that the COPS subset is more constrained with n e > n/2 for 
all the problems. Another feature of the COPS subset is that the equality constraints are 
the result of either difference or collocation approximations to differential equations. 

We ran our final complete runs with the same options for all models. The options 
involve setting the output level for each solver so that we can gather the data we need, 
increasing the iteration limits as much as allowed, and increasing the super-basics limits 
for MINOS and SNOPT to 5000. None of the failures we record in the final trials 
include any solver error messages about having violated these limits. While we relieved 
restrictions unnecessary for our testing, all other parameters were left at the solvers' 
default settings. 

The script for generating the timing data sends a problem to each solver succes- 
sively, so as to minimize the effect of fluctuation in the machine load. The script tracks 
the wall-clock time from the start of the solve, killing any process that runs 3,600 sec- 
onds, which we declare unsuccessful, and beginning the next solve. We cycle through all 
the problems, recording the wall-clock time as well as the combination of AMPL sys- 
tem time (to interpret the model and compute varying amounts of derivative information 
required by each solver) and AMPL solver time for each model variation. We repeat the 
cycle for any model for which one of the solvers' AMPL time and the wall-clock time 
differ by more than 10 percent. To further ensure consistency, we have verified that the 
AMPL time results we present could be reproduced to within 10 percent accuracy. All 
computations were done on a SparcULTRA2 running Solaris 7. 

We have ignored the effects of the stopping criteria and the memory requirements 
of the solvers. Ideally we would have used the same stopping criteria, but this is not 
possible in the AMPL environment. In any case, differences in computing time due to 
the stopping criteria are not likely to change computing times by more than a factor of 
two. Memory requirements can also play an important role. In particular, solvers that 
use direct linear equation solvers are often more efficient in terms of computing time 
provided there is enough memory. 

The solvers that we benchmark have different requirements. MINOS and SNOPT 
use only first-order information, while LANCELOT and LOQO need second-order in- 
formation. The use of second-order information can reduce the number of iterations, but 
the cost per iteration usually increases. In addition, obtaining second-order information 
is more costly and may not even be possible. MINOS and SNOPT are specifically de- 
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signed for problems with a modest number of degrees of freedom, while this is not the 
case for LANCELOT and LOQO. As an example of comparing solvers with similar 
requirements, Section[6]shows the performance of linear programming solvers. 

4. Case Study: Optimal Control and Parameter Estimation Problems 

We now examine the performance of LANCELOT 0, MINOS ED, SNOPT El, and 
LOQO 1 18 1 on the subset of the optimal control and parameter estimation problems in 
the COPS |7| test set. Figures |4~T1 and l4!2l show the performance profiles in different 
ranges to reflect various areas of interest. Our purpose is to show how performance pro- 
files provide objective information for analysis of a large test set. Figure l4TT1 shows the 

Performance Profile on Subset of COPS 
1 1 1 1 1 1 1 1 1 1 

0.9 - 




LANCELOT 
LOQO 
MINOS 
SNOPT 



0.1 - 



1 i _i i i i i i i 

1 23456789 10 

X 

Fig. 4.1. Performance profile on [0, 10] 

performance profiles of the four solvers for small values of t. By showing the ratios 
of solver times, we eliminate any weight of importance that taking straight time differ- 
ences might give to the problems that require a long run time of every solver. We find 
no need to eliminate any test problems from discussion. For this reason, solvers receive 
their due credit for completing problems for which one or more of the other solvers 
fails. In particular, 1 — Ps{t) is the fraction of problems that the solver cannot solve 
within a factor r of the best solver, including problems for which the solver in question 
fails. 
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Performance Profile on Subset of COPS 




100 



Fig. 4.2. Performance profile on [0, 100] 



From this figure it is clear that LOQO has the most wins (has the highest probability 
of being the optimal solver) and that the probability that LOQO is the winner on a given 
problem is about .61. If we choose being within a factor of 4 of the best solver as the 
scope of our interest, then either LOQO or MINOS would suffice; but the performance 
profile shows that the probability that these two solvers can solve a job within a factor 
4 of the best solver is only about 70%. SNOPT has a lower number of wins than either 
LOQO or MINOS, but its performance becomes much more competitive if we extend 
our t of interest to 7. 

Figure l4~2l shows the performance profiles for all the solvers in the interval [1, 100]. 
If we are interested in the solver that can solve 75% of the problems with the greatest 
efficiency, then MINOS stands out. If we hold to more stringent probabilities of com- 
pleting a solve successfully, then SNOPT captures our attention with its ability to solve 
over 90% of this COPS subset, as displayed by the height of its performance profile for 
t > 40. This graph displays the potential for large discrepancies in the performance ra- 
tios on a substantial percentage of the problems. Another point of interest is that LOQO, 
MINOS, and SNOPT each have the best probability p s (r) for r in some interval, with 
similar performance in the interval [15, 40]. 

An observation that emerges from these figures is the lack of consistency in quartile 
values of time ratios. The top three solvers share a minimum ratio of 1, and LOQO 
and MINOS also share first quartile values of 1. In other words, these two solvers are 
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the best solvers on at least 25% of the problems. LOQO bests MINOS's median value 
with 1 compared with 2.4, but MINOS comes back with a third quartile ratio of 4.3 
versus 13.9 for LOQO, with SNOPT mixing results further by also beating LOQO with 
12.6. By looking at Figures |4~T1 and l4~2l we see that progress between quartiles does 
not necessarily proceed linearly; hence, we really lose information if we do not provide 
the full data. Also, the maximum ratio would be Tm for our testing, and no obvious 
alternative value exists. As an alternative to providing only quartile values, however, 
the performance profile yields much more information about a solver's strengths and 
weaknesses. 

We have seen that at least two graphs may be needed to examine the performance 
of the solvers. Even extending r to 100, we fail to capture the complete performance 
data for LANCELOT and LOQO. As a final option, we display a log scale of the per- 
formance profiles. In this way, we can show all activity that takes place with r < tm 
and grasp the full implications of our test data regarding the solvers' probability of suc- 
cessfully handling a problem. Since we are also interested in the behavior for r close to 
unity, we use a base of 2 for the scale. In other words, we plot 

t i ► —size {p eP : log 2 (r PtS ) < r} 

Tip 

in Figure 14.31 This graph reveals all the features of the previous two graphs and thus 
captures the performance of all the solvers. The disadvantage is that the interpretation 
of the graph is not as intuitive, since we are using a log scale. 

Figures l4~Tl and l4~2l are mapped into a new scale to reflect all data, requiring at least 
the interval [0, log 2 (1043)] in Figure l4~3l to include the largest r p s < r^j. We extend 
the range slightly to show the flatlining of all solvers. The new figure contains all the 
information of the other two figures and, in addition, shows that each of the solvers fails 
on at least 8% of the problems. This is not an unreasonable performance for the COPS 
test set because these problems were generally chosen to be difficult. 

5. Case Study: The Full COPS 

We now expand our analysis of the data to include all the problems in version 2.0 of 
the COPS |7| test set. We present in Figure lSTl a log2 scaled view of the performance 
profiles for the solvers on that test set. 

Figure ISTl gives a clear indication of the relative performance of each solver. As in 
the performance profiles in Section^ this figure shows that performance profiles elim- 
inate the undue influence of a small number of problems on the benchmarking process 
and the sensitivity of the results associated with the ranking of solvers. In addition, per- 
formance profiles provide an estimate of the expected performance difference between 
solvers. 

The most significant aspect of Figure l5~T1 as compared with Figure H~3l is that on 
this test set LOQO dominates all other solvers: the performance profile for LOQO lies 
above all others for all performance ratios. The interpretation of the results in Figure l5~TI 
is important. In particular, these results do not imply that LOQO is faster on every 
problem. They indicate only that, for any r > 1, LOQO solves more problems within 
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Fig. 4.3. Performance profile in a log 2 scale 



a factor of r of any other solver time. Moreover, by examining p s (l) and p s {tm), we 
can also say that LOQO is the fastest solver on approximately 58% of the problems, 
and that LOQO solves the most problems (about 87%) to optimality. 

The difference between the results in Section|4]and these results is due to a number 
of factors. First of all, as can be seen in Table 13. ll the degrees of freedom for the full 
COPS test set is much larger than for the restricted subset of optimal control and parame- 
ter estimation problems. Since, as noted in Section|5] MINOS and SNOPT are designed 
for problems with a modest number of degrees of freedom, we should expect the per- 
formance of MINOS and SNOPT to deteriorate on the full COPS set. This deterioration 
can be seen by comparing Figure |5~T1 with Figure l4~3l and noting that the performance 
profiles of MINOS and SNOPT are similar but generally lower in Figure lSTl 

Another reason for the difference between the results in Section0]and these results 
is that MINOS and SNOPT use only first-order information, while LOQO uses second- 
order information. The benefit of using second-order information usually increases as 
the number of variables increases, so this is another factor that benefits LOQO. 

The results in this section underscore our observation that performance profiles pro- 
vide a convenient tool for comparing and evaluating the performance of optimization 
solvers, but, like all tools, performance profiles must be used with care. A performance 
profile reflects the performance only on the data being used, and thus it is important to 
understand the test set and the solvers used in the benchmark. 
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6. Case Study: Linear Programming 

Performance profiles can be used to compare the performance of two solvers, but per- 
formance profiles are most useful in comparing several solvers. Because large amounts 
of data are generated in these situations, trends in performance are often difficult to see. 
As a case study, we use data obtained by Mittelmann 1 14|. Figure [O] shows a plot of 
the performance profile for the time ratios in the data Benchmark of LP solvers on a 
Linux-PC (5-25-2000), which includes results for COPLJLP ( 1 .0), PCx (1.1), SOPLEX 
(1.1), LPABO (5.6), MOSEK (1.0b), BPMPD (2.11), and BPMPD (2.14). 

In keeping with our graphing practices with the COPS set, we designate as failures 
those solves that are marked in the original table as stopping close to the final solution 
without convergence under the solver's stopping criteria. One feature we see in the 
graph of Mittelmann's results that does not appear in the COPS graphs is the visual 
display of solvers that never flatline. In other words, the solvers that climb off the graph 
are those that solve all of the test problems successfully. As with Figure 1431 all of the 
events in the data fit into this log-scaled representation. While this data set cannot be 
universally representative of benchmarking results by any means, it does show that our 
reporting technique is applicable beyond our own results. 

As in the case studies in Sections|4]and[5] the results in Figure l6~Tl give an indication 
of the performance of LP solvers only on the data set used to generate these results. In 
particular, the test set used to generate Figure loTl includes only problems selected by 
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Log 2 Scaled Performance Profile for LP Solvers 




o 2 4 6 8 10 12 ...r 



Fig. 6.1. Performance profile for linear programming solvers 

Mittelmann for his benchmark. The advantage of these results is that, unlike the solvers 
in Sections0]and|5] all solvers in Figure RTTl have the same requirements. 

7. Conclusions 

We have shown that performance profiles combine the best features of other tools for 
benchmarking and comparing optimization solvers. Clearly, the use of performance 
profiles is not restricted to optimization solvers and can be used to compare solvers in 
other areas. 

We have not addressed the issue of how to select a collection of test problems to 
justify performance claims. Instead, we have provided a tool - performance profiles - 
for evaluating the performance of two or more solvers on a given set of test problems. 
If the data is obtained by following careful guidelines 1811 31 . then performance profiles 
can be used to justify performance claims. We emphasize that claims on the relative 
performance of the solvers on problems not in the test set should be made with care. 

The Perl script perf . pi on the COPS site |7| generates performance profiles for- 
matted as Matlab commands to produce a composite graph as in Fi gures |4~TI and R~2l 
The script accepts data for several solvers and plots the performance profile on an in- 
terval calculated to show the full area of activity. The area displayed and scale of the 
graph can then be adjusted within Matlab to reflect particular benchmarking interests. 
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